IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Application 

Inventor(s): Patrick Chiu et al. 

Apptn. No.; 10/636,044 

Confirm, No.: 9973 

Filed: August 8, 2003 

Title: PEER TO PEER GESTURE BASED 



Examiner: Nicholas Augustine 
Atty. Docket No.: FXPL-0I060USO 



Art Unit: 2179 



PATENT APPLICATION 



MODULAR PRESENTATION SYSTEM 



Customer No. 23910 



DECLARATION OF INVENTORS UNDER 37 CF.R. § 1.131 



I. We the undersigned, Patrick Chiu, Qiong Liu and John Steven Boreczky, declare as 
follows: 

2 We are employees of FX Palo Alto Laboratories and specialize in computer software 
research and development 

3. We are the inventors of the invention described and claimed in the above U.S. patent 
application. 

4. For convenience, embodiments of this invention hereafter are referred to as the 
u ModSlideShow". 

5. For convenience, Claim 1 of the above application is set out below with elements (a)-(c) 
identified as follows: 

A system for providing content in a modular presentation system, comprising: 

(a) a plurality of displays, wherein each display neighbors at least one other display and at 
least two of the plurality of displays are in visual proximity to each other; 

(b) an input device that receives input of a gesture to move a content from a first display of 
the plurality of displays; and 

(c) a processor, that interprets a direction to move the content from the First display based 
on the gesture, that specifies a second display to which the content is to be moved, based on 
the gesture and the position of the plurality of displays and that propagates the content of 
the first display to the second display. 

6. A prototype of the k% ModSlideShow° which for convenience we identify as 
"ModSlideShowl" was reduced to practice on or before April 18, 2002. 

7. The "ModSlideShowP prototype was implemented with a rear projector and touch screen 
plasma displays. 
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8. The "ModSlideShowT prototype was implemented with laptops connected on a wireless 
network, where a location sensing system can detect the room location of a laptop. 

9. The u ModSlideShow 1 " prototype integrated a digital compass with a laptop. 

10. The "ModSlideShowl" prototype contained all the elements of Claim I as described and 
claimed in the above U.S. patent application, 

11. The "ModSIideShowT prototype comprises a plurality of displays, wherein each display 
neighbors at least one other display and at least two of the plurality of displays are in visual 
proximity to each other as claimed in Claim 1, element (a). 

12. The "ModSlideShowr' prototype comprises an input device that receives input of a gesture 
to move a content from a first display of the plurality of displays as claimed in Claim 1, 
element (b), 

13. The "ModSlideShowr' prototype comprises a processor that interprets a direction to move 
the content from the first display based on the gesture, that specifies a second display to 
which the content is to be moved, based on the gesture and the position of the plurality of 
displays and that propagates the content of the first display to the second display, as 
claimed in Claim 1 , element (c). 

14. For convenience, Claim 5 of the above application is set out below with elements (d)-(g) 
identified as follows: 

A method of providing content in a modular presentation system having a plurality of displays, 
wherein at least two of the plurality of displays are, the method comprising: 
(d) receiving input of a gesture to move a first content presented on a first display of the 
plurality of displays; 

(c) interpreting a direction to move the content from the first display based on the gesture; 

(f) specifying a second display to which the first content is to be moved based on the gesture 
and the relative position of the plurality of displays; and 

(g) presenting the first content at the second display and a second content at the first display. 

15. The "ModSlideShowr* prototype contained all the elements of Claim 5 as described and 
claimed in the above U.S. patent application. 

16. The "ModSlideShowr' prototype is able to receive input of a gesture to move a first 
content presented on a first display of the plurality of displays as claimed in Claim 5, 
element (d). 
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17. The "ModSIideShowl" prototype is able to interpret a direction to move the content from 
the first display based on the gesture as claimed in Claim 5, element (e). 

18. The "ModSIideShowT prototype is able to specify a second display to which the first 
content is to be moved based on the gesture and the relative position of the plurality of 
displays as claimed in Claim 5, element (0- 

19. The u ModSlideShowl" is able to present the first content at the second display and a 
second content at the first display as claimed in Claim 5, element (g). 

20. For convenience, Claim 1 3 of the above application is set out below with elements (h)-(j) 
identified as follows: 

A computer readable medium with instructions for execution by a computer for providing 
content in a modular presentation system having a plurality of displays* wherein at least 
two of the plurality of displays are in physical and visual proximity to each other, the 
instructions comprising: 

(h) receiving input of a gesture to move first content presented on a first display; 

(i) interpreting a direction to move the content from the first display based on the gesture; 
and 

(j) presenting the first content at the second display and a second content at the first 
display. 

21. The "ModSlideShowl" prototype contained all the elements of Claim 13 as described and 
claimed in the above U.S. patent application. 

22. The "ModSlideShowl" prototype is able to receive input of a gesture to move first content 
presented on a first display as claimed in Claim 13, element (h). 

23. The "ModSlideShowP prototype is able to interpret a direction to move the content from 
the first display based on the gesture as claimed in Claim 13, element (i). 

24. The * 4 ModSlidcShowt" prototype is able to present the first content at the second display 
and a second content at the first display as claimed in Claim 1 3, .element (j), 

25. A report on the "ModSlideShow: A Slide Presentation System for Peer to Peer Modular 
Displays with a Gestural Interaction Technique" was prepared on or before April 18, 2002. 

26. The report "ModSlideShow: A Slide Presentation System for Peer to Peer Modular 
Displays with a Gestural Interaction Technique" is shown in a redacted version as Exhibit 
<A\ 
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27. We signed the report on "ModSlideShow: A Slide Presentation System for Peer to Peer 
Modular Displays with a Gestural Interaction Technique" between the dates of April 18, 
2002 and April 22, 2002. 

28. A second prototype of the "ModSlideShow" which for convenience we identify as 
"ModSlideShow2" was reduced to practice on or before April 30, 2002, 

29. The u ModSlideShow2" prototype included all the features of the "ModSlideShowT 
prototype. 

30. The "ModSlideShow2" prototype was implemented with a rear projector and touch screen 
plasma displays. 

31. The i4 ModSIideShow2" prototype was implemented with laptops connected on a wireless 
network, where a location sensing system can detect the room location of a laptop. 

32. The u ModSlideShow2" prototype integrated a digital compass with a laptop. 

33. The "ModSlideShow2" prototype contained all the elements of Claim 1 as described and 
claimed in Che above U.S. patent application. 

34. The u ModSlideShow2" prototype comprises a plurality of displays, wherein each display 
neighbors at least one other display and at least two of the plurality of displays are in visual 
proximity to each other as claimed in Claim I, element (a). 

35. The "ModSlideShow2" prototype comprises an input device that receives input of a gesture 
to move a content from a first display of the plurality of displays as claimed in Claim I, 
element (b). 

36. The it ModSlideShow2" prototype comprises a processor that interprets a direction to move 
the content from the first display based on the gesture, that specifies a second display to 
which the content is to be moved, based on the gesture and the position of the plurality of 
displays and that propagates the content of the first display to the second display, as 
claimed in Claim 1, element (c). 

37. The "ModSlideShow2" prototype contained all the elements of Claim 5 as described and 
claimed in the above U.S. patent application. 

38. The "ModSlideShow2" prototype is able to receive input of a gesture to move a first 
content presented on a first display of the plurality of displays as claimed in Claim 5, 
element (d). 

39. The u ModSIideShow2 ,t prototype is able to interpret a direction to move the content from 
the first display based on the gesture as claimed in Claim 5, element (e). 
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40. The "ModSlideShow2" prototype is able to specify a second display to which the first 
content is to be moved based on the gesture and the relative position of the plurality of 
displays as claimed in Claim 5, element (f). 



4L The "ModSlideShow2" is able to present the first content at the second display and a 
second content at the first display as claimed in Claim 5, element (g). 

42. The "ModSlideShow2" prototype contained all the elements of Claim 13 as described and 
claimed in the above U.S. patent application. 

43. The "ModSlideShow2" prototype is able to receive input of a gesture to move first content 
presented on a first display as claimed in Claim 13, element (h). 

44. The "ModSlideShow2" prototype is able to interpret a direction to move the content from 
the first display based on the gesture as claimed in Claim 13, element (i). 

45. The u ModSltdeShow2" prototype is able to present the first content at the second display 
and a second content at the first display as claimed in Claim 13, element (j). 

46. The "ModSlideShow2" prototype included the command propagation feature. 

47. The date of creation of computer files that make up different modules involved in the 
"ModSlideShow2" prototype is shown in Exhibit B" 

48. The date of creation of the ModSlideShow.java file shown in Exhibit ET was April 26, 
2002. 

49. The date of creation of the GestRecognizer.class file shown in Exhibit B" was April 30, 
2002, 

50. The "ModSlideShowP and "ModSIideShow2" prototypes were designed and constructed * 
at the FX Palo Alto Laboratories, Palo Alto, California, United States of America. 

51. The "ModSlidcShowl" and u ModSlideShow2 M prototypes were demonstrated at the FX 
Palo Alto Laboratories, conference room, in Palo Alto, California, United States of 
America. 

52. The "ModSlideShowl" and "ModS!ideShow2" prototypes were demonstrated to an 
internal FX Palo Alto Laboratories research and development team on or before March 25, 
2003. 
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53. The undersigned, being hereby warned that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under 18 U.S.C §1001, and that such 
willful false statements may jeopardize the validity of the application or any resulting 
registration, declares that the facts set forth in this declaration are true; all statements made 
of his/her own knowledge are true; and all statements made on information and belief are 
believed to be true. 



Date Patrick Chiu 

Date Qiong Liu 

Date Johi/Steven Boreczky 
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Proposal Title: ModSlideShow: A Slide Presentation System for Peer-to- 
Peer Modular Displays with a Gestural Interaction Technique 

Brief Description: U is becoming increasingly common for meeting rooms to be equipped 
with multiple wall displays. There are also non-fixed displays such as those on laptops or 
handhelds that are brought into the meeting room by the presenters or other participants. We 
describe a slide presentation system called ModSlideShow for showing and manipulating slides 
on a heterogeneous set of displays. This system provides a modular structure for the various 
displays and links them together basedjsnjocation adjacency and compass orjeniatipn; these 
attributes may be set manually for fixed wall displays and dynamically sensed for mobil e 
displays. A gestural interaction technique enables slides to be pushed or pulled in a particular 
direction/orientation — with the slides propagating along that direction — and promotes intuitive 
and fluid interaction. For teleconferences, virtual directions and orientation s are supported. 
Gestures may be perform ed on a vvaTf display or a mobile computer's display using a touch-panel 
on top of the display, or alternatively using a pointing device such as a pen, mouse, or laser 
pointer. 
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Description of Invention 



Introduction 

Presenting slides is an integral parr of many meetings. The current standard setup is to use a laptop or 
meeting room computer to show PowerPoint slides or Web-bases slides on a single video projector. 
! Becoming increasingly common and affordable are wall displays with rear projectors and large plasma 

displays (see f igure \ '). Some of these displays, like modular furniture (see Figure 5), can be rearranged for 
different types of meetings. In the near future, many meeting rooms will contain a heterogeneous set of 
various displays including video projectors, plasma panels, displays on tabletops, displays on laptops or 
notebook computers, and displays on handheld devices. Some of these displays allow the user to interact 
directly with them, while others merely show the video monitor output of the computer driving it. More 
generally, a meeting setup may connect a distributed set of meeting rooms and their displays as in a 
teleconference, 

In this IP, we describe a slide presentation system called ModSlideShow for showing and manipulating 
slides on a heterogeneous set of displays. This system provides a modular structure for the various displays 
and links them together based on location adjacency and orientation. For fixed large wall displays, the 
location and orientation attributes may be set manually, while for mobile displays (e.g. laptops), the 
attributes may be dynamically sensed. A number of research location sensing systems have been developed 
(e.g. see ( I J, f20), [21 J), with ultrasonic systems giving sufficient accuracy (resolution of 9cm). Compass 
orientationTi^beWTectcd with a digital compa ss [41 More advanced ultrasonic systems [10] claim to do 
both position and angle with very high precision (position accuracy of 1 .5 mm RMS and angular accuracy 
of 0.05 degree RMS ). For teleconferences, virtual directions and orientations can be defined. The various 
displays in the ModSlideShow system form a peer-to-peer network that communicate with one another as 
Web services. The Web services infrastructure is highly suitable for a modular system with heterogeneous 
components. 

In ModSlideShow, the metaphor for the set of presentation slides is a stack of slide images. Each display 
holds a stack, and the top slide of each stack is shown. A slide is always shown "full screen". Slides can be 
passed around from display to display by manipulating the stacks of the respective displays. 

A gestural interaction technique in ModSlideShow promotes intuitive, fluid, and efficient slide 
manipulation. We have designed a set of simple gestures that can be made with the finger directly on a 
touch screen of a plasma panel, with a stylus on a notebook or handheld computer, or with a mouse or other 
pointing device on a laptop. Alternatively, remote devices such as laser pointers or 3D mice may be used. . 
It is also possible to employ computer vision to detect gestures (e.g. see [6], (24)). 

Many of the gesture commands have a direction component. A unique feature of ModSlideShow is 
command propagation in some direction along a series of adjacent displays. The displays are configured so 
that each display has only a small number of adjacent displays, typically determined by location and 
orientation relat ionships. This configuration is essentially a subgraph of the peer-to- peer network. 
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Figure 1 . Center display is a rear video projector, the two side displays are plasma 
displays with touch screens, plus a laptop* pen notebook, and handheld computers 
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Figure 2. On the right wall display, a line-shaped gesture toward the left pushes its slide to 
its left (the center display), and command propagation on the center display pushes its 
slide to the display on its left (the leftmost display) 




Figure 3. A single gesture can be used to send the next 3 slides in the laptop display s 
stack to the 3 wall displays 
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Figure 4. Four plasma displays mounted together on a wall as a mosaic 




Figure 5. Plasma display with touch screen on a cart 

We give two concrete examples to illustrate how ModSHdcShow works. Suppose in Figure I, the presenter 
starts with a stack of slides on the display on the far right behind the podium. To advance to the next slide, 
the presenter makes a straight-line gesture, motioning from right to left, on the touch-screen of the display. 
This is interpreted by the system as a command to push the slide to the display on its right (our left). By 
allowing commands to propagate, the display on the right sends its top slide to the display on its right. See 
Figure 2. Upon reaching the last wall display (on our far left), the incoming slide covers the top slide on 
this display's stack. The result has the desirable effect that the most recent 3 slides are always shown, and 
each slide operation is accomplished with a single gesture command. 
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In our second example, we illustrate how a single gesture can also invoke a transaction with several slides 
sent to several displays. Suppose in Figure I ( we have a laptop and three wall displays with a stack of slides 
on the laptop- A straight upward line gesture made on the laptop (interpreted in this case as "put up next 
hunch of slides") sends the next three slides up to the three displays all at once. See Figure 3. Like the 
previous example, die user makes far fewer operations than would otherwise be required if each slide were 
to be moved manually from display to display. 

We note that more generally, the techniques described in this IP can be used to move other kinds of content 
besides presentation slide images around displays. Since what is ac tual ly t ^i ^ncd jjjj ffl L > tnat URL 
can be a pointer to any kind of content: video, Web pages, etc. 



This IP is organized as follows, in the next two .sections we describe more fully the gesture set and the 
effects of the gesture commands. Then we describe some scenarios of use, technical details, novelty, and 
related work. 

Basic Gesture Set 

Basically, a user can give a display a PUSH or PULL command, along with other parameters including the 
direction: R (right), L (left), U (up), D (down), plus the four diagonal directions; UR (up right), UL (up 
left), DR (down right), DL (down left). Another parameter is speed, which is calculated by how rapid the 
gesture motion is made. 

A push gesturojsa simple line stroke along with a direction. We specify a line gesture by its direction: 
Qj^recUon (e.g. G_R for a line gesture to the right, G_UL for a diagonal line gesture to the upper left). 

A pull gesture is a do uble-line stro ke made by going in one direction and corning back. We specify a 
dou^fe'Tme gStuTe'by its two successive directions: GJtireaionl direction! (e.g. G_R.-L for double-line 
gesture right-left, GJJR-DL for a gesture to the upper right ami hack). 

These gestures are simple, intuitive in terms of expressing a direction, easy for users to make and easy for 
the system to recognize. 



Effects of Gesture Commands 

After the user injn rtgjtgssturc, the syjtejiy nte rnrets the gesture a s a PUSH or PULL command along with 



At any given time, sotne of the displays have a set of slides while other displays do not. in the preferred 
design, slides are shown full-screen without visible UI widgets. Slides may bgj fflsdfifl^ 
doing the equivalent ojQijrj^ up a File menu. Bach display's set of slides is a 

stack of images. 

A PUSH command moves the slide that is displayed to another display along the direction of the gesture. 
Usually this is the adjacent display; however, when the speed of the gesture isjugh, the slide can skim along 
(and the animated effect is visible) Uu^ougjije^ The slide is removcJTrom the top of the 

source display's stack and passed on, and the next slide in the stack is displayed. On the target display, the 
received slide goes to the top of its stack. A PULL command is the inverse of the PUSH command. 
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An option. for PUSH/PULL is whether or not to transport a slide or a copy of the t s lide. For example, a 
"member of the audience may pull a d uglicaifi copy o f the slide on a wall display onto his laptop display 
without disrupting the presentation. Each display can set permissions ja jspecify wh ich other displays can 
pull a slide or copy of a slide from it. 

A display in the modular structure can have up to eight adjacent displays in the eight directions listed in the 
previous section. The con figuration am be setmanua lly (e.g. withjlxeiby^ or d ynamically 

sensed (e.g. with mobile TapTop displays). The configuration may also be set for a particular session (e.g. a 
teleconference). Displays are registered by name and IP address and can be referred to by one another for 
command transactions. 

For each adjacent display, an QptianutDiyJte„M.i 0 ?^ ow commands to propagate. For example, when a 
display receives a PUSH command to the right along wittPallideTu command and 

push its curre ntly displayed slide to the r ight, and then display the recejyjalk ide. Typically, it makes sense 
for shared wall displays to all gvy propagation and personal ^ to allow it. For shared wall 

displays, prc^a^ncm^provides a way to keep visible the most recent A' slides (iV being the number of wall 
displays) with minimal effort as the presenter steps through the slides by gesturing on the leftmost (or 
rightmost display. On the odier hand, for personal mobile displays, propagation is not desirable because 
having slides shuttle around would be distracting. Both parameters, the direction and speed, are 
propagated. 

A display can also conduct co mm and transact ions. A command transaction is a command that executes a 
s^ofsu'bjSMtands, A useful" transaction pushes (or jjuUsjthe next /V slides on the display's stack (e.g. N 
being the number of wall displays). This enablesltie user to do a presentation by updating all the wall 
displays together with a single UP gesture made on her laptop. 

Another possibility is to use a transaction to splk up a slide image into M&jjo^ j ^ 
display mosaic (see Figure 4) to construct a larger image. 

A more passive feature is enable a display to mirron^^ always obtaining and showing a copy 

of the slide as it is put up on the origuiallftsplay. This "allows audience members and remote sites to obtain 
a copy of the slide. 

For security and access control, displays may also be set to not accept commands from other displays, and 
more sophisticated schemes including ones similar to group permissions can be employed. 

Scenarios of Use 

To provide some motivation for the design of the gesture command set and the interaction among the 
modular displays, we list several scenarios of use. These scenarios are also summarized in Table 1 below. 



Witnessed and Understood by _ JgSkV^ Date tj/l&fo 

1 . Submitter Signature '^tft^ C*=± ° Me - ? ' (t — X 

2. Submitter Signature ^ Oate_Mllll/^k. 

3. Submitter Signature Zg*& Date ±^L^ 1 



Exhibit A 





Scenario 


# of 

wall displays 


f*nf 

mobile displays 


n of 

presenters 


Tasks, features 


1 


Presentation 


N> 1 


0 


i 


Step thru recent N slides, 
••throw'* slides 


? 


Presenter with laptop 


N> 1 


1 

(presenter) 


i 


Presenter moves laptop 
around, sense orientation of 
laptop 


'A 


Discussion, 
design meeting 


N> 1 


N> I 

(participants) 


N> 1 

(using mobile di plays) 


Put up slide, share slide, 
sense location & orientation 
of mobile displays 




Presentation with 
active audience 


N> I 


N> 1 
(audience) 


I 


Take snapshot of slide, 
annotate, put up for Q & A 


>5 


Teleconference 


N> 1 


N> 1 


1 or 
N> 1 


Mirror displays at remote site 



Table 1. Scenarios of use 

Scenario i : Fix ed Wall D isplays Only, w ith One Prescmer 

We also assume thai at least one of the wall displays has a touch screen for gesture input. In this case, due 
to the displays being all fixed, the modular structure and adjacency relations among the displays can be 
manually set (so dynamically determining the location and orientation is not required). In a typical 
presentation, the presenter creates a set of slides (a directory of JPG or GIF images, which is easily 
produced by applications such as PowerPoint with its Save As command). The slides are loaded onto one 
of the displays by using the Fife Open command under the context menu activated with a right-lap on the 
display. 

In the setup with 3 displays shown in Figure U one way to show the slides is for the presenter to stand at the 
rightmost (from our point of view) display near the podium. As the presenter steps through the slides by 
making push gestures toward the right, the previously shown slides flow from the rightmost display to the 
leftmost display, with the most recent 3 slides remaining visible. To go bac k a slide jhc j?re a 
left-righi^ttiLge^^^ **P S trough mc $ct 

of slides, the slides accumulate on the leftmost display. See Figure 2. 

Another operation in the Figure I setup is to push a slide from the right plasma display to the left plasma 
display without covering the slide in the large center display. This is done by performing left gesture with a 
fast motion, which is interpreted as a gesture with high speed. The result is that the slide image skims from 
the right display to the left display, showing up on the center display momentarily for animation effect, 

Scen ario 2: Fixed Wall Dis play s, with One Presenter and hi s Mobile Display 

This is similar to scenario I , except that the presenter inputs ges ture commands on the laptop's display. 
The laptop sets its adjacent displays either manually by the presenter or automatically using location and 
orientation sensing technology. Since the speaker faces the audience, the laptop display faces the wall 
displays behind her. Thus, the orientation of the controlling display is opposite to that in scenario I . By 
takmgjuu^^ the system interprets the gesture directions correctly and the 

slides end up where the speaker would expect. 
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A variation is making use of all 3 displays by showing 3 slides at a time, The command transaction option 
on the presenter's laptop is set to send the next 3 slides to the 3 wall displays. During the presentation, the 
presenter can make a PUSH UP gesture to invoke this command. See Figure 3. 



S cenario 3: Fixed Wall Displays, with Multi ple People and their Mobile Displays 

This scenario is typical of discussions and design meetings in which a number of people will show a slide 
from their laptops and handholds on the large wall display for discussion. There are some interesting 
possibilities for interaction, which are described in the following. 

A user can make a PUSH UP gesture on his laptop to show a slide (or a copy of it) up on a wall display. 
The display assoc iatcd with the UP direction may bemanually set, or dynamically sen se d if the laptop has 
location and orientation sensors. In the latter case, the"iSeT7oTates his laptop so that its display and the 
desired wall display face roughly the same direction. To transport the slide to the display on the right of its 
UP display, the user makes a diagonal gesture toward the upper right direction. 

A user can make a PUSH RIGHT gesture to mo ^^b£j^ c from his laptop to his neighbor's laptop on the 
right. Another wayTcT transfer slides between mobile displays is for one user to PULL a sli de to h is laptop 
when another person shows that slide on a wall display. 

For sharing slides directly among the laptops* it is natural for two people to move physically near each 
other. When their laptops are in close proximity, the gesture directions and targets can be accurately 
interpreted and used to transfer slides. More generally, a menu command can be used to send a slide to any 
registered local or remote display. 

Scenario 4: Fixed Wall Displays , wit h On e Presenter, plus Mult i ple People and thei r Mobile Displays 
This scenario is like scenario I or 2, with audience members also carrying mobile displays. A user in the 
a 5i£!i£ nce ca n ..£ et a ^P^ lMil^g^^^ai^ is being presented on the wall display. A user in the audience 
may jpuff in a copy o f a sli de from a walLdispfav, annotate Jton his pen computer, and put it hack ufcpn a 
wall display for asking question about the highlighted builet on the slide. In order to avoid too much 
disruption during a presentation, not all of the dis plays s ho ul d he set t o allo w thejiudience tQjiiauLsUdes to. 
Also, while it is less likely for the presentejr tojrans pon. a slide to a member of the audience, the 
functionality is there. 

Sce nario 5: Fixed Wall Displays and Mobile Displays in a Teleconference 

The general teleconference scenario has similarities to the above scenarios. One difference is that it is 
desirable to use the mirroring feature and have the slides on the wall displays (or the main large wall display 
as in the center of Figure 1 ) shown simultaneously at the remote sites. 

Technical Details 

Each display runs a n instan ce of the ModSlideShow software, ModSlideShow is a Java application/applet. 
The instances of ModSlideShow communicate with one another via J<Mki^ : - which is'a WcCTServfces 
standard. By using the Web Services infrastructure, the system can connect displays in distribute d locations 
for teleconferences. 

The slide collections are l oaded and reside on Web serve rs, and the URLs to tjiejiio^^images are passed 
around rather than the image files. Since noj>bjects aj^passed around, a tigTuv^ej^hj[j>ri)tocol such as 
XMLRPC is suitable for this application. When non-fixed displays are supported, aregjsiry forthe 
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discovering the adjacent display is required; th is can be a s im ple listing or handled withj LWch services 
mgistrj;. Other issues such as security and transactions can also leverage the technology for handling these 
In thcTemerging Web services infrastructure. 

Since what is actually transported is a URL, other kinds of content can be transported around the displays 
besides presentation slide images. The URL can be a pointer to any kind of content: video, Web pages, etc. 

Different types of hajr^^ejUje^^ gesture in put. Fingers can be used on wall displays with 

touch screen^luch as the SMART Board for Plasma Displays { 18]. Some displays (e.g. video projectors) 
rtmfnolluprx)rt gesture input, but they can still show slides and propagate gesture commands from adjacent 
displays. A^sjxiM^s use.djb^ on mobile pen comp uters such as notebooks and handheld*. 

Mouse or ^ooHcr p^oTallr^dcvice can be used to input gestures on a laptop. There are also laptops with 
touch screens. Also, remote L controU[ey ices such as laser pointers or 3D mice may be used with both wall 
displays and mobile displays. 

The gestures used in ModSlideShow are simple line shapes. These are easily recognized by looking at 
.simple geometric features of the gesture path such as the width and height of the bounding box, and the 
N location of the starting and ending points. The speed attribute is calculated by looking at the time 
difference between the endpoints of the gesture, 
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LilcSlideShow [3], a predecessor to ModSlideShow. supports transporting a copy of a slide to a remote 
display for teleconferences. This has been used with great success for over a year for teleconferences 
between Palo Alto and Japan. 

An early prototype ModSlideShow has been implemented for the FXPAL conference room with a rear 
projector and touch-screen plasma displays (see Figures I and 2). This prototype also worts with laptops 
hooked up on the wireless network. A location sensing system that can delect the room location of a laptop 
has been deployed (UCLA system [ ] ]), a finer resolution system such as (21] is not yet deployed. We have 
integrated a digital compass [4| with a laptop (i.e. a mobile display). Emerging Web services standards for 
security, registry, and transactions have not yet been integrated into ModSlideShow. 
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Others known to have worked on this or a similar invention 

See section below on related work. 

Related invention proposals, patents, publications or commercial work 

The simplest systems for using multiple displays merely show on a set of displays the output of a single 
computer. Current presentation systems typically use a laptop with a video output signal that shows a copy 
of the laptop's display on a video projector or a large wall display. In the Unix operating system, a display 
can he set to view a particular workstation that can be located anywhere on the network. More recently, 
popular technology such as VNC (Virtual Network Computing) and Microsoft Terminal Services Client 
allow one computer's display and keyboard/mouse input to control another computer. 

Software for application sharing allows the contents of a shared workspace to be viewed on distributed 
displays, along with supporting user interactions with the shared application. Such software systems 
include Microsoft NetMeeting. Xerox MectingBoard and Tivoli [151, and Smart Technologies M-Path [181. 
With application sharing, the content shown on one display is similar to that on the other displays. 

Smart Technologies SynchronEycs [18J connects a teacher's PC display (which may be projected on a wall) 
with a number of students' PC displays. The teacher's display can broadcast to all the students' displays, 
and can also show any student's display to the entire class. Only one slide is shown at a time during a 
presentation, and the system is not peer-to-peer. 

Pebbles SlideShow Commander [ 13] supports using a PDA to control a PowerPoint presentation on a 
laptop. The laptop's video output is hooked up in a standard way to the room's wall display. The PDA 
shows a thumbnail of the slide, and the PDA's buttons are used for moving forward/backward among slides. 
The presenter has a private view thai includes notes to the slides. 

A bunch of displays can be tiled together to create a large hi-resolution display. An example is the 
Princeton Scalable Display Wall [I 1 1. Since the resulting tiled display is intended to behave like a single 
large display, the spirit of such systems is very different from our loosely coupled modular displays. 



11 




Dene HJJ&I AX 



Dm , 4-'f-6 V 

Date *£/]l/kh - 

Date. Ur**'<* 



Exhibit A 



More sophisticated systems thai allow separaie display units to interact with each other are closer to our 
work. ConnecTabUss [19] supports the dynamic coupling of displays for small tables with a display on top. 
Two tables must be physically moved together in ordered for them to be coupled. Only two displays can be 
connected at a time, front edge to front edge. The two displays can either form a single surface or remain as 
two separate surfaces, and the two users can work on computer applications and share data. 

Large displays that support gestural input on the screen include the Smart Board and Smart Board for 
Plasma Displays [J 8). Earlier research work on electronic whiteboards includes LiveBoard [5| and its 
Tivoli software 1 12). Tivoli uses gestures for manipulating content on a whiteboard-sized display surface (a 
copy of the work surface may be shown via application sharing explained above). For larger wall-sized 
displays, an interaction technique developed at GMD [9] uses gestures to manipulate objects on a single 
surface; it has a ''throw" gesture for moving an object across the display, and a "shuffle" gesture for moving 
a single object a by a distance equal to its length. Another project at Stanford [22] explores using gestures 
made by a laser pointer on a single large continuous display. 

Several interaction techniques were invented at Sony for moving data between displays. Pick-and-drop ( 16| 
is an interaction technique for moving data between displays by directly "picking" up an object from one 
display with a pointing device and "dropping" it on another display. Picking and dropping involves 
touching the displays with a pen, so location and orientation of the displays are irrelevant. Pick-and drop 
affects only the displays touched by the pen, and actions are not propagated to other displays. Another 
technique called hyperd ragging [17] extends the reach of a cursor seamlessly from a portable laptop's 
display to the displays (projected) on the tables and walls in an augmented environment, and objects can be 
<}rtig^ y mi and dropped among different displays and computers. With hyperdragging, the displays are 
modeled as one continuous surface for the cursor to navigate in. Location and orientation of tagged laptops 
are detected with video cameras and are used to determine the edges of the laptop's display to determine the 
placement of the cursor as it jumps to another display. 

While our main scenario is characterized by direct manipulation, with the user touching the display's screen 
or using the display computer's input device to perform push and pull gestures on the presentation slides, it 
is also possible to make the gestures with a separate remote control device or laser pointer. In FXPAL-IP- 
01-018 18), a hand-held remote control device is proposed for controlling multiple audio, video and other 
displays and peripherals. It is interfaced with a media switching system and the user can "select" and 
"paste" among the desired displays or output devices. In one of the embodiments, a six-degree- of- freedom 
pointer device that has location and orientation tracking is employed. A technique for handheld computers 
at CMU called "semantic snarling" (I3J enables the meaning of a screen's contents, rather than a 
screenshot, to be copied to the handheld (e.g. a snarfed menu is not just an image of the menu buta 
functioning menu). This mechanism allows cutting and pasting among displays. Some earlier work at MIT, 
Spatial Data-Management System (SDMS) [1), uses wall display and two monitors next to the armrests of a 
chair. One of the monitors shows an overview of the data; the user can select some small portion of data 
and the selected data is displayed in a more detail view on the large screen. SDMS also explored using 
speech recognition with a position-orientation Polhemus device to enable a user to draw a shape on the wall 
display (e.g. by pointing and saying "Create a blue circle there."). 

Infrastructure for pccr~to~pccr software modules for controlling multimedia devices has been explored in 
the Medusa system [23]. The hyperdragging system 1 17] uses lava and RMI (remote method invocation) to 
manage data between computers (which are modeled as one continuous interaction or surface). Our 
modular peer-to-peer software model is simpler and more easily integrate set of heterogeneous displays 
because ours is based on the Web Services infrastructure standards. 
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Has invention been built, made, run, or tested? 

Sec Status section. 

Is the invention used in a current produces ) or planned for use in a future 
product(s)? 

ModSlidcShow technology will likely be used and/or licensed in FX Smart Spaces products. 

Dates of any previous or planned future disclosures external to Xerox 

UIST02, (November 2002), or later in CHI '03 (April 2003), precise dates for these conferences have not 
been finalized. 

Source of outside funding 

None. 
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